add initial cargo ci #3409

Kasama · 2025-10-15T17:02:12Z

Description of Changes

API and ABI breaking changes

Expected complexity level and risk

Testing

bfops · 2025-10-15T17:27:24Z

The tests are failing. I think several of them just need a Cargo.lock update, but at least one seems to be failing for an unknown subcommand.

.github/workflows/ci.yml

tools/ci/src/main.rs

bfops · 2025-10-24T17:16:14Z

tools/ci/src/main.rs

+            run!("for p in \"$GITHUB_WORKSPACE\" \"${RUNNER_TEMP:-/__t}\" \"${RUNNER_TOOL_CACHE:-/__t}\"; do [ -d \"$p\" ] && setfacl -R -m u:ue4:rwX -m d:u:ue4:rwX \"$p\" || true; done")?;
+
+            run!("export CARGO_HOME=\"${RUNNER_TOOL_CACHE:-/__t}/cargo\"")?;
+            run!("export RUSTUP_HOME=\"${RUNNER_TOOL_CACHE:-/__t}/rustup\"")?;


FYI I think a lot of this doesn't apply to users running these tests locally

bfops · 2025-10-24T17:21:52Z

tools/ci/src/main.rs

+
+        Some(CiCmd::CliDocs) => {
+            run!("pnpm install --recursive")?;
+            run!("cargo run --features markdown-docs -p spacetimedb-cli > docs/docs/cli-reference.md")?;


note that this assumes we're running from the root directory

it does. Do you think this it a problem to assume that? If so, would it be a good idea to try to figure out where in the dir tree the command was ran (maybe looking upwards for the root Cargo.toml or something?) to adjust accordingly?

I might just add a check similar to what tools/upgrade-version does:

tools/upgrade-version/src/main.rs- let current_dir = env::current_dir().expect("No current directory!"); tools/upgrade-version/src/main.rs- let dir_name = current_dir.file_name().expect("No current directory!"); tools/upgrade-version/src/main.rs- if dir_name != "SpacetimeDB" && dir_name != "public" { tools/upgrade-version/src/main.rs: anyhow::bail!("You must execute this binary from inside of the SpacetimeDB directory, or use --spacetime-path");

(alternatively we could use git rev-parse --show-toplevel and then just go to that directory)

.github/workflows/ci.yml

bfops · 2025-10-24T17:29:28Z

.github/workflows/ci.yml

+        run: cargo ci self-docs --check
+
  cli_docs:
    name: Check CLI docs


I assume we want to replace this one with a cargo ci invocation as well?

bfops · 2025-10-24T17:31:05Z

tools/ci/build.rs

@@ -0,0 +1,4 @@
+#[allow(clippy::disallowed_macros)]
+fn main() {
+    println!("cargo:rustc-env=TARGET={}", std::env::var("TARGET").unwrap_or_default());


what's this part for?

this allows for the default parameter for target on the update-flow. Mostly for running it locally without needing to specify that flag manually.

SpacetimeDB/tools/ci/src/main.rs

Line 177 in 1c0406d

let target = target.unwrap_or_else(|| env!("TARGET").to_string());

ah gotcha. in that case, nit: we could either use the variables that are provided already:

const ARCH: &str = env!("CARGO_CFG_TARGET_ARCH"); const OS: &str = env!("CARGO_CFG_TARGET_OS"); const ENV: &str = env!("CARGO_CFG_TARGET_ENV");

or we could only pass --target {target} to cargo build if it's provided by the user.

I kind of prefer that last option. It seems intuitive still.

(honestly I'm not 100% sure that we even need the target param.. I think the default on each platform might be the one we want anyway.. but we should keep it for now just for parity with the original CI).

now that --target is optional, is it reasonable to remove this?

bfops · 2025-10-24T17:38:26Z

tools/ci/src/main.rs

+    let mut env = env::vars().collect::<HashMap<_, _>>();
+    env.extend(additional_env.iter().map(|(k, v)| (k.to_string(), v.to_string())));
+    log::debug!("$ {cmdline}");
+    let status = cmd!("bash", "-lc", cmdline).full_env(env).run()?;


FYI I'm not 100% sure how this will work on windows

I'm not sure how to work this out, actually. Because all other commands are also unix-like. Do you have an idea on how to make it work on windows too?

I'd say we ask @jdetter to try running some of cargo ci on his windows machine. If it "just works" then great. If it doesn't due to bash missing, then I think we might need to make a separate ticket for making all of our CI properly runnable on windows. I suspect it might work properly in the GitHub workflow as-is though, because I believe that will have bash installed.

bfops · 2025-10-24T17:46:08Z

tools/ci/src/ci_docs.rs

+
+use crate::Cli;
+
+pub fn generate_cli_docs() -> String {


nit: any chance we could use clap_markdown like we do for the SpacetimeDB CLI docs?

indeed. I didn't notice that we had that

since this is already working, we could add a TODO and just leave it for a future PR.

…/cargo-ci

bfops · 2025-10-27T16:57:00Z

FYI I pushed a merge commit so the CI would run again

…d/cargo-ci

bfops · 2025-10-29T17:10:16Z

.github/workflows/ci.yml

      - name: Run smoketests
-        # Note: clear_database and replication only work in private
-        run: python -m smoketests ${{ matrix.smoketest_args }} -x clear_database replication
+        run: cargo ci smoketests -- ${{ matrix.smoketest_args }}


little nit: I kinda preferred when the skipped tests were provided here, since they're going to be skipped on every platform. Not super important, but also not super important to change (although we could consider renaming smoketests_args to extra_smoketest_args to be clearer).

If you prefer it this way, that's fine with me.

bfops · 2025-10-29T17:12:15Z

tools/ci/src/main.rs

@@ -162,19 +161,20 @@ fn main() -> Result<()> {
            run!("cargo run -p spacetimedb-cli -- build --project-path modules/module-test")?;
        }

-        Some(CiCmd::Smoketests { args }) => {
+        Some(CiCmd::Smoketests { mut args }) => {
+            let default_args = ["-x", "clear_database", "replication"];


nit: I worry about tests being omitted by default. I think the expectation was that there's some easy/intuitive way to run everything.

bfops · 2025-10-29T17:18:40Z

tools/ci/src/main.rs

+
+        Some(CiCmd::CliDocs) => {
+            run!("pnpm install --recursive")?;
+            run!("cargo run --features markdown-docs -p spacetimedb-cli > docs/docs/cli-reference.md")?;


I might just add a check similar to what tools/upgrade-version does:

tools/upgrade-version/src/main.rs- let current_dir = env::current_dir().expect("No current directory!"); tools/upgrade-version/src/main.rs- let dir_name = current_dir.file_name().expect("No current directory!"); tools/upgrade-version/src/main.rs- if dir_name != "SpacetimeDB" && dir_name != "public" { tools/upgrade-version/src/main.rs: anyhow::bail!("You must execute this binary from inside of the SpacetimeDB directory, or use --spacetime-path");

(alternatively we could use git rev-parse --show-toplevel and then just go to that directory)

bfops · 2025-10-29T17:29:57Z

tools/ci/src/main.rs

nit: if we use .map().unwrap_or_default() above, we could use something similar here:

Suggested change

let github_token_auth_flag = github_token_auth.then_some("--features github-token-auth ").unwrap_or_default();

I don't feel strongly one way or another.

(or we could have an extra_args string that we append to if target is defined or github_token_auth is true)

bfops · 2025-10-29T17:31:44Z

tools/ci/build.rs

@@ -0,0 +1,4 @@
+#[allow(clippy::disallowed_macros)]
+fn main() {
+    println!("cargo:rustc-env=TARGET={}", std::env::var("TARGET").unwrap_or_default());


now that --target is optional, is it reasonable to remove this?

bfops · 2025-10-29T17:35:34Z

tools/ci/src/main.rs

+    let mut env = env::vars().collect::<HashMap<_, _>>();
+    env.extend(additional_env.iter().map(|(k, v)| (k.to_string(), v.to_string())));
+    log::debug!("$ {cmdline}");
+    let status = cmd!("bash", "-lc", cmdline).full_env(env).run()?;


I'd say we ask @jdetter to try running some of cargo ci on his windows machine. If it "just works" then great. If it doesn't due to bash missing, then I think we might need to make a separate ticket for making all of our CI properly runnable on windows. I suspect it might work properly in the GitHub workflow as-is though, because I believe that will have bash installed.

Kasama added 2 commits October 13, 2025 14:17

add initial cargo ci

4824f5f

merge master into add/cargo-ci

c8581e2

bfops self-requested a review October 15, 2025 17:26

bfops added release-any To be landed in any release window no runtime change This change does not affect the final binaries labels Oct 20, 2025

update cargo ci

1c0406d

bfops reviewed Oct 24, 2025

View reviewed changes

[add/cargo-ci]: Merge remote-tracking branch 'origin/master' into add…

c5ea361

…/cargo-ci

Kasama added 4 commits October 28, 2025 21:36

Merge branch 'master' of github.com:clockworklabs/SpacetimeDB into ad…

b86cce7

…d/cargo-ci

fix: PR comments

852c13c

update ci to call cargo for unreal test

2b2250a

update cargo ci docs

a9b46b2

bfops reviewed Oct 29, 2025

View reviewed changes


	let github_token_auth_flag = github_token_auth.then_some("--features github-token-auth ").unwrap_or_default();

add initial cargo ci #3409

Are you sure you want to change the base?

add initial cargo ci #3409

Uh oh!

Conversation

Kasama commented Oct 15, 2025 • edited by bfops Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Changes

API and ABI breaking changes

Expected complexity level and risk

Testing

Uh oh!

bfops commented Oct 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bfops Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bfops commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Kasama commented Oct 15, 2025 •

edited by bfops

Loading

bfops Oct 24, 2025 •

edited

Loading